Catch trial performance

## [1] "Excluded 31 participants based on catch-trial performance."

Exclusion of random guesses

We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.

## [1] "Excluded 1 participants based on random responses."

Aggregated results

Individual responses

AUC computation

We use the AUC function with the splines method to directly compute the AUC.

t-test and regression model with control variables:

## 
##  Two Sample t-test
## 
## data:  aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = 2.9171, df = 142, p-value = 0.004108
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##   4.110558 21.393403
## sample estimates:
## mean of x mean of y 
## 18.011474  5.259493
## 
## Call:
## lm(formula = auc_diff ~ cond + test_order + first_speaker_type + 
##     confident_speaker, data = rbind(aucs.cautious, aucs.confident))
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -75.675 -13.841   0.159  16.143  72.807 
## 
## Coefficients:
##                                  Estimate Std. Error t value Pr(>|t|)   
## (Intercept)                        15.530      4.879   3.183  0.00180 **
## condconfident (probably-biased)   -12.752      4.387  -2.907  0.00425 **
## test_orderreverse                  -2.275      4.392  -0.518  0.60527   
## first_speaker_typeconfidentfirst    5.716      4.395   1.300  0.19562   
## confident_speakerconfidentm         1.209      4.395   0.275  0.78372   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 26.32 on 139 degrees of freedom
## Multiple R-squared:  0.06997,    Adjusted R-squared:  0.04321 
## F-statistic: 2.615 on 4 and 139 DF,  p-value: 0.03789